Skip to content

Conversation

@hanishkvc
Copy link
Contributor

@hanishkvc hanishkvc commented Nov 10, 2025

This adds the initial basic skeleton for supporting vision models going forward to the tools/server/public_simplechat alternate web client ui. Basic tested with Gemma3 and Qwen3VL

This builds on the previous PR in this series ie #17038 which completed the current initial go at tool calling which also includes a set of client side based builtin and bundled tool calls for code execution, data storage, web access, web search, pdf and xml use, ... as well as support for looking into reasoning/chainofthought of the models which share the same.

also possible refinement wrt trapping, if needed, added as comment

all or allSettled to use or not is the question.

whether to wait for a round trip through the related event loop or
not is also a question.
Dont forget to map members of got entity from fetch to things
from saved original promise, bcas remember what is got is a promise.

also

add some comments around certain decisions and needed exploration
Declare the result of UrlReq as a DataClass, so that one doesnt
goof up wrt updating and accessing members.

Duplicate UrlRaw into UrlText, need to add Text extracting from
html next for UrlText
As _UrlopenRet not exposed for use outside urllib, so decode and
encode the data.

Add skeleton to try get the html/xml tree top elements
As html can be malformed, xml ElementTree XMLParser cant handle
the same properly, so switch to the HtmlParser helper class that is
provided by python and try extend it.

Currently a minimal skeleton to just start it out, which captures
only the body contents.
First identify lines which have only whitespace and replace them
with lines with only newline char in them.

Next strip out adjacent lines, if they have only newlines
Ensures that if the url being requested as any query strings in
them then things dont get messed up, when the url to get inc its
query is extracted from the proxy request's query string
also now track header, footer and nav so that they arent captured
Add a new send headers common helper and use the same wrt the
overridden send_error as well as do_OPTIONS

This ensures that if there is any error during proxy opertions,
the send_error propogates to the fetch from any browser properly
without browser intercepting it with a CORS error
ie opening db as well as a transaction to access a store within
the db.
Added put and get helpers wrt indexedDB.

Updated save and load related logics in SimpleChatTCRV.
Allow passing the accept list for file type input element helper.

Inturn given that currently it is used for the image selection
for vision models, set it to jpeg and png in the caller for the
same.
@hanishkvc
Copy link
Contributor Author

Changed Chat session auto save and user optional load to use indexedDB instead of localStorage, so that there is enough space availabe for saving chat sessions, given that images are in the mix now.

Cleaned up the ui a tiny little bit

@hanishkvc hanishkvc changed the title server/public_simplechat vision (wip), toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) server/public_simplechat vision, toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) Nov 12, 2025
@hanishkvc hanishkvc changed the title server/public_simplechat vision, toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) server/public_simplechat vision (basic ok), toolcall (done, with 0 setup clientside builtin tools+), reasoing(done) Nov 12, 2025
Had forgotten to move the one shot resp into try catch before.
Fixed it. Ensure both oneshot multipart resp try catch

Add some todos for later.

Add a new check wrt response being normal or a error related one
ie the content is actually a error message.
Trap the mouseenter and mouseleave events wrt the chatmessage
block to show the chatmessage-popover ui inturn within the
corresponding chatmessage.

When ever user clicks the popover menu's delete button, the
uniqId associated with each chat message, is now used to
delete the user selected chat message.

Initial skeleton wrt clipboard copy.
Use position fixed followed by using position-area so that one
can use anchor passed through showPopover.

Avoid the hidePopover from mouseLeave, because as the mouse moves
over the poped up popover ui, the mouseLeave will be called on
the chat message block/container. And that would have closed the
popover and inturn mouseEnter would have got called on ChatMsg,
given now with the popover hidden, the ChatMsg block is directly
accessible. So this leads to twinkling / toggling ie continues
show/hide of the popover.

Inturn in mouseEnter itself decide when to close the previous
popover (ie if the chatmessage id is different from what it was
before).
As position-area is not yet officially supported in firefox (its
only in nightly builds, as of now), so switching to the inset
block/inline start/end css properties.
Currently only the text/content part is handled wrt copy. Image
URL isnt handled, even if present.
Rename to get_chatmessage_index.

Return the index of the chatmessage deleted.

Add a scroll element into view helper.

Now the UniqId wrt the ChatMessage is stored as a attribute
wrt the corresponding chatmessage block. And same used wrt
scrolling after deleting messages.

When a chat message is deleted, previously the screen would have
moved to last message in the chat session, now it tries to remain
wrt adjacent messages on the screen.
This ensures that the tool call edit/trigger is reshown, when user
submits a empty response and the logic stops from going ahead.
avoid refreshing the full chat session messages ui

also Update readme
If deleting a non last (and non just before last) message, then
just directly remove the corresponding chat message block from the
ChatSession UI and be done with it.

However if deleting the last (or just before last) message, then
one needs to decide on whether tool call edit/trigger ui is shown
or removed and so... and similarly wrt tool response edit/submit.
Make the toolresp non happy path use a direct insert chat msg
wrt ui, rather than recreating the full chat session ui fresh.
Rename lastMsg to prevLastMsg to match semantic better

Return seperate flags to indicate add and show success or not

Update tool response happy path to use this optimised add show
instead of chat_show (which recreates the full chat session ui).
Now even the user response handling flow (handle_user_submit) has
been updated to use the optimised chat show logic which adjusts
/ updates the chat session ui wrt the last two messages only and
not the full chat history in the chat session.

Move the logic wrt last two messages and ui, into its own func
and inturn use it everywhere as required.
Needed as some places we require to delete the chat message as
well as update the corresponding ui block.

And in other places we require to just remove the ui block from
chat session ui and not the underlying chat message from chat history.

Change ui updateprev appendlast to use the ui remove logic.
SC.add discards any temp role message and not just tool temp msg

New SC.add_smart which either adds a new message to the chat session
or replaces existing last message in the chat session, based on same
role or different, given that the chat session cant have the same
role type chat message occuring adjacent in it. Inturn rename MCUI
chatmsg_add_uishow to chatmsg_addsmart_uishow and use add_smart

* helps wrt trying to rerun a tool call with modified args or so.
  Rather prev discard temp role messages in SC.add not good enough
  as uniqId will change in its case.
  Helps avoid adding duplicate ToolTemp messages in chat session ui

* can help if loading a prev chat session which ended in a user
  message without a ai response. User can type in a new message
  and continue that old chat session, with the new message, replacing
  the old user message as well as initiating handshake with ai server
  in a proper manner.

Replace MCUI chatmsg_ui_updateprev_appendlast with chatmsg_ui_refresh
which is a more generic flow, which takes care of updating the ui as
needed irrespective of if the specified set of messages are already
in the chat session ui or not. Also allows the caller to control how
many messages at the end need to be refreshed wrt ui.
Rename chatmsg_ui_refresh to chat_uirefresh.

Even in case of the delete path and inturn deleting one of the last
two messages in a chat session, now use the generified chat uirefresh
logic instead of the chat_show full session refresh / recreation of
the session ui

Inturn to make the uirefresh really generic and usable in all cases
including above case, now take care of clearing the tool call edit
/ trigger at the beginning, so that the last 2 messages decide in
turn whether to show a tool call edit/trigger ui or not, as well as
the tool call response edit / submit ui.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python python script changes server

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant